Multi-agent training of a color identification neural network

ABSTRACT

Various embodiments of the systems and methods described herein are directed towards training an artificial neural network to identify color values of a sample by providing image data obtained through multiple image capture devices under a plurality of lighting conditions. The present invention also includes using a pre-trained neural network to identify the color values of a sample having an unknown color value by capturing an image of an unknown color sample and known color reference samples under any illumination or hardware configuration.

FIELD OF THE INVENTION

The present invention is directed to systems and methods for identifyingcolors present in an image of a color sample where the images are takenusing unknown camera parameters and under unknown illuminants.

BACKGROUND OF THE INVENTION

There is often a need to capture the color of a sample and determine theclosest match for the color of that sample from an existing database.For example, a user may want to find out the color of a wall or piece offurniture and search through the paint manufacturer's fan deck toidentify the color. As an alternative to a tedious manual search, acolorimeter or a spectrophotometer is a more sophisticated tool toidentify a color. However, such devices are both expensive andinconvenient to use. Furthermore, such specialty devices are unlikely toaccompany a user on trips, scouting events or excursions.

Efforts have been made to use images taken with a camera to select colorfrom a database. For example, U.S. Pat. No. 8,634,640 to Nina Bhatti et.al. (herein incorporated by reference as if presented in its entirety)teaches a method to select a color palette from a camera image, aidedwith a reference color chart. In that method, matrix transformations areused to convert images taken with an unknown illuminant to knownilluminant, and thus eliminate the need of controlled illumination.However, since matrix transformations are linear operations, the methodtaught by Bhatti has strict requirements for the hardware used by thesystem. Additionally, the system taught by Bhatti is sensitive to noise,resulting in diminished performance under a wide range of operationalconditions.

It has been known in the art to use Artificial Neural Networks (ANNs) toassist in color recognition. For example, U.S. Pat. No. 5,907,629, toBrian Vincent Funt et. al. (herein incorporated by reference as ifpresented in its entirety) teaches a method of estimating thechromaticity of illumination of a colored image consisting of aplurality of color-encoded pixels with a pre-trained neural network.However, such a system does not solve the problem associated withdetermining the color value of an unknown subject using a pre-trainedNeural Network.

Thus, it will be helpful to have a system and a method that can utilizeavailable cameras such as those in smartphones, to eliminate the need ofcontrolled illumination, and have improved tolerance of nonlinearity andnoise, learn and adapt to large user data sets. Therefore, what isneeded in the art are systems, methods and computer program products forevaluating color values and searching for the same or variouscombinations thereof.

SUMMARY OF THE INVENTION

Embodiments of the present invention are directed towards systems,methods and computer program products for training an artificial neuralnetwork to identify the color of samples having unknown color values. Inone particular embodiment, a plurality of image capture devices(multiple image capture agents) are used to capture images of samplesand a color reference chart comprised of color reflectance samples underone of a plurality of known illuminants. Both the elements of the colorreference chart and sample have a known triple of color values under areference illuminant (such as CIE D65). Here, each image capture deviceis configured to transmit the captured image to a processor or adatabase. One or more processors having memory and configured by programcode executed thereby, receive the transmitted images and extract fromthe transmitted images the color values of the pixels corresponding toarea of the image depicting the sample color and each of the colorreference or reflectance elements. The extracted color values are usedto train the neural network by applying the extracted color values for agiven sample and reference elements as input nodes in an input layer ofan artificial neural network and assigning the known CIE color valuesfor the sample color as the output nodes of the artificial neuralnetwork.

In an alternative configuration, the present invention is directed tosystems and methods of using a pre-trained artificial neural network toestimate or identify the CIE color values of an unknown color sample.Here, an image of a reference chart, such as one having a center portion(e.g., a hole through which a test color sample is viewed) and aplurality of identical color reflectance element groups are captured bya capture device. A processor having memory is configured by programcode executed thereby to extract RGB color values from the pixelsrepresenting the center portion of the sample reference chart. Likewise,the processor is configured to extract RGB color values for each of theplurality of color reference elements. The processor further providesthe extracted sample RGB color values and plurality of reference RGBcolor values as an input to one or more input nodes of an input layer ofa neural network pre-trained to identify colors. The neural networkincludes an output layer containing output nodes. In one particularimplementation, the output nodes correspond to the CIELAB (CIE colorcomponents) values. The artificial neural network also includes at leastone intermediate layer having a plurality of nodes connectible betweenthe input and output layers. The processor, through the use of theneural network, generates at the respective output node, the colorvalues that characterize color values of the sample color with the userilluminant rectified to a reference illuminant. Theseuser-illuminant-independent CIE color values are compared to the entriesof a color value database that use the reference illuminant. Bysearching the color value database, the color most closely matching thesample color is identified and provided to a user.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is illustrated in the figures of the accompanying drawingswhich are meant to be exemplary and not limiting, in which likereferences are intended to refer to like or corresponding parts, and inwhich:

FIG. 1A illustrates the color reference chart according to oneembodiment of the present invention.

FIG. 1B illustrates an alternative view of the color reference chartaccording to one embodiment of the present invention.

FIG. 2 illustrates a block diagram of a color search utilizing a neuralnetwork according to one embodiment of the present invention.

FIG. 3 presents a flow diagram detailing the steps taken in oneembodiment of the color searching system according to one embodiment ofthe present invention.

FIG. 4 presents a block diagram detailing specific components of thecolor searching system according to one embodiment of the presentinvention

FIG. 5 presents a graph diagram detailing a representative node layerarrangement of an artificial neural network used to identify colorvalues according to one embodiment of the present invention.

DETAILED DESCRIPTION OF CERTAIN EMBODIMENTS OF THE INVENTION

By way of overview, various embodiments of the systems and methodsdescribed herein are directed towards using a pre-trained neural networkto identify the CIE color values of a color sample in an image capturedby an image capture device. The captured image includes both the samplecolor under investigation, as well as a number of color referenceelements. A pre-trained neural network is used to evaluate the measuredRGB values of the pixels corresponding to the depiction of a samplefound within the image. Using an artificial neural network eliminatesthe need to capture the image under specific illuminants, orientationsor with particular devices in order to obtain the CIE color coordinatesof the sample.

Color Reference Chart

A color reference chart (as provided in FIGS. 1A-B) is used to bracketthe unknown color sample with a collection of known color values. Asshown in FIG. 1, the color reference chart 102 includes a number ofdiscrete color reference elements or reflectance samples. Here, thecolor reference chart 102 is divided into four (4) quadrants about acenter portion 106. In one or more configurations, the center portion isan aperture or transparent window permitting the observation of aportion of a sample 104 when the color reference chart 102 is placedin-between the image capture device 202 and the sample 104.Alternatively, the center portion 106 is a stage or platform suitablefor the placement of a sample 104 under analysis on the platform.

The four (4) quadrants of the color reference chart each contain thesame number of reflectance samples. In a particular arrangement, thequadrants are arranged to be four-fold rotationally symmetric about thecenter portion 106. In the described arrangement, each reflectanceelement found within a given quadrant will have counterparts in theother quadrants. Thus, while the possible number of element groups (herereferred to as quadrants) are based on particular design constraints(e.g. fourfold symmetry or threefold symmetry), the number ofreflectance elements within each group will always be the same.Furthermore, each member of a group of color reflectance elements hascounterpart reflectance elements in each of the remaining colorreference groups. It should be noted that while each counterpart colorelement will have approximately the same wavelength-dependentreflectance value, variations in production of reflectance elementsmeans that each counterpart reflectance sample will have some deviationsfrom the known or “true” wavelength-dependent reflectance value for thatreflectance element.

With reference to the specific configuration of the color referencechart, a color reference chart having four-fold symmetry presents aneffective way to ensure that each section of the color reference charthas same number of elements and that the shape of the color elements canbe easily tiled into the area occupied by that group without overlap orempty space. Furthermore, fourfold rotation symmetry allows thecounterpart reflectance elements to be at four positions underpotentially spatially non-uniform illumination. Thus, by averaging thecamera values determined for a reflectance element and its matchingcounterparts, a RGB color value can be estimated that is representativeof the camera values obtained if the reflectance element had been in thecenter portion 106.

Those possessing an ordinary level of requisite skill in the art willappreciate that other configurations, such as the reference elementshaving threefold symmetry about the sample, can be used in accordanceaims described herein.

With reference to FIG. 1B, quadrants A-D each are rotated 90 degreesrelative to the neighboring quadrants. For example, Quadrant Crepresents a 90 degree counter-clockwise rotation of each element withinthe quadrant using the center of the reference chart as the pivot point.As shown in particular detail and described herein, each corner elementof the color reference chart 102 is an identical color. Due to four-foldrotational symmetry, the color directly below the corner element inQuadrant A, is the same color as the element directly to the right ofthe corner element in Quadrant C.

In one particular embodiment, the provided color reference chartincludes 45 color reference elements. Thus, where there are fourquadrants, the color reference chart provides at least 180 colorreferences for use in connection with the artificial neural networksystem. However, other groups and numbers of colors are also possibleand envisioned.

Sample

In one arrangement, the sample 104 captured by the image capture deviceis a color fan, or color fan deck providing a range of known colorsvalues. As described in more detail herein, known CIE color value datais used in combination with measured RGB data from the image capturedevice in order to train the artificial neural network. The sample 104can also be any object where the CIE color value of the object isunknown or in need of clarification. For example, where a fan deck ofcolors has been used to train the neural network, an object having anunknown color value can be matched to one of the colors in the fan deck.

Image Capture Device

With general reference to FIG. 2, the color reference chart 102 isplaced in-front of a sample 104 in need of analysis. As an example, thecolor reference chart 102 is placed over a sample 104 having a paintcolor of particular interest. An image capture device 202 is used tocapture an image of both the image reference chart 102 and the color ofthe sample 104, such as by capturing the color of the sample 104 throughaperture. Upon acquisition of the image, the image capture device 202transmits the captured image to a processor 204. The image capturedevice 202 transmits the image as a binary stream, bit string, pixelarray, file or collection of files or other suitable format for theprocessor 204 to evaluate.

Where the image capture device is used to acquire an image of a colorsample having known CIE values, a collection of camera types orconfigurations can be used. For instance, multiple images of the samesample can be captured using different image capture devices, such asdifferent smart-phone cameras. Likewise, similar make and model imagecapture devices can be used under different illuminants to capture acollection images having slight variation in measured RGB color valuesdue to differences in hardware.

In a particular arrangement, the image capture device 202 is one or morecameras or image acquisition devices such as CMOS (Complementary MetalOxide Semiconductor), CCD (charged coupled device) or other imageacquisition devices and associated hardware, firmware and software. Theimage capture device 202, in accordance with one embodiment, isintegrated into a smartphone, tablet, cell phone, or other portablecomputing apparatus. In a further embodiment, the image capture device104 is an “off the shelf” digital camera or web-camera that is connectedto the processor 204 using standard interfaces such as USB, FIREWIRE,Wi-Fi, Bluetooth, and other wired or wireless communication technologiessuitable for the transmission image data.

Computer or Processor

The image obtained by the image capture device 202 is transmitted to oneor more processor(s) 204 for evaluation. The processor 204 is configuredthrough one or more modules to determine corrected color values for thesample color based on an analysis of the image color values by apre-trained neural network. The corrected color value is used to searchrepository, such as a database 108 or other storage location containinga listing of color values and associated names or labels and output theresults to a user through a display 210.

With further reference to FIG. 2, the processor 204 is a computingdevice, such as a commercially available cellphone, smartphone, notebookor desktop computer configured to directly, or through a communicationlinkage, receive images captured by the image capture device 202. Theprocessor 204 is configured with code executing therein to accessvarious peripheral devices and network interfaces. For instance, theprocessor 204 is configured to communicate over the Internet with one ormore remote servers, computers, peripherals or other hardware usingstandard or custom communication protocols and settings (e.g., TCP/IP,etc.).

In one configuration, the processor 204 is a portable computing devicesuch as an Apple iPad/iPhone® or Android® device or other commerciallyavailable mobile electronic device executing a commercially available orcustom operating system, e.g., MICROSOFT WINDOWS, APPLE OSX, UNIX orLinux based operating system implementations. In other embodiments, theprocessor 204 is, or includes, custom or non-standard hardware, firmwareor software configurations. For instance, the processor 204 comprisesone or more of a collection of micro-computing elements,computer-on-chip, home entertainment consoles, media players, set-topboxes, prototyping devices or “hobby” computing elements. The processor204 can comprise a single processor, multiple discrete processors, amulti-core processor, or other type of processor(s) known to those ofskill in the art, depending on the particular embodiment.

In one or more embodiments, the processor 204 is directly or indirectlyconnected to one or more memory storage devices (memories) to form amicrocontroller structure. The memory is a persistent or non-persistentstorage device (such as memory 205) that is operative to store theoperating system in addition to one or more of software modules 207. Inaccordance with one or more embodiments, the memory comprises one ormore volatile and non-volatile memories, such as Read Only Memory(“ROM”), Random Access Memory (“RAM”), Electrically ErasableProgrammable Read-Only Memory (“EEPROM”), Phase Change Memory (“PCM”),Single In-line Memory (“SIMM”), Dual In-line Memory (“DIMM”) or othermemory types. Such memories can be fixed or removable, as is known tothose of ordinary skill in the art, such as through the use of removablemedia cards or modules. In one or more embodiments, the memory of theprocessor 204 provides for the storage of application program and datafiles. One or more memories provide program code that the processor 204reads and executes upon receipt of a start, or initiation signal. Thecomputer memories may also comprise secondary computer memory, such asmagnetic or optical disk drives or flash memory, that provide long termstorage of data in a manner similar to the persistent memory device 205.In one or more embodiments, the memory 205 of the processor 204 providesfor storage of application program 207 and data files when needed.

Acquiring the Image Data

As shown with reference to FIGS. 3-4, the processor 204 is configured bycode stored in the memory. Such code includes one or more softwaremodules that configure the processor 204 to instruct the image capturedevice 202 to capture an image of the color reference chart and sample104. For example, the processor 204 is configured by an image capturemodule 402 to capture the image as provided in step 302 of the work flowof FIG. 3. The captured image includes pixel data that corresponds tothe color of the sample under investigation as well as color informationfor each of the reference elements arranged about the aperture of thereference chart 102.

The color value extraction module 404 configures the processor 204 toextract RGB color values from the pixels corresponding to the colorreference elements as well as pixels corresponding to the aperture.According to step 304, the processor 204 extracts the color informationby one or more image analysis procedures implemented as sub-modules ofthe color value extraction module 404.

In one non-limiting example, the processor 204 is configured by one ormore image analysis submodules to identify features of interest withinthe image. Here, the features identified within the image are theindividual reflectance elements and the aperture of the color referencechart from the acquired image. Upon identification of the relevantfeatures in the image, the processor 204 extracts color valuescorresponding to each reference element. As an example, the average RGBvalues for the area corresponding to a particular reference element arecalculated and stored as the measured color value for a given locationwithin the image. Here, the image analysis sub module evaluates therelative difference in pixel color values between each corner elementand the color elements in direct contact with the corner element. In theillustration provided in FIG. 1, the two reference elements in directcontact with the corner elements have different hues. The arrangement ofthe elements, as determined on hue difference provides sufficientinformation to determine the orientation of the quadrants.

Where the color reference chart contains two or more identical colorelements distributed (such as four elements in the four foldrotationally symmetric arrangement) the processor 204 can be configuredto average the measured color values for the materially equivalentsamples to determine an average measured color value for a specificcolor found in the color reference elements. By averaging the measuredvalues for each of the elements having the same known color value, theaveraged measurement is used by the processor 204 to compensate spatialinhomogeneity of the illuminant and obtain an estimation of the RGBcolor values for the reflectance sample as if it were placed in thecenter portion 106. However, other arrangements, such as selecting apreferred color reference element as representative of a specific colorvalue are also envisioned.

In a further arrangement, the color values extracted according to step304 are RGB color values. Upon extraction of the color values for thesample and reference elements, the color values are stored in the localor remote memory 205. Using the color reference chart of FIG. 1B, foreach image processed by the processor 204 configured by the colorextraction module 404, a data object or objects referencing the RGBreading of the sample color and the 180 reference colors are generated.

In one particular implementation, the RGB readings of the sample and thereflectance elements are transformed or manipulated in order to formatthe values for use in the artificial neural network. For example, anormalization submodule of the color value extraction module 404normalizes the RGB values extracted to the range of [0, 1] prior tofurther processing. However, in other arrangements, the RBG valuesobtained from the images are stored in the original format or form.

Those skilled in the art will appreciate for the foregoing that theextracted RGB color values can be used in accordance with the presentinvention to either train a neural network to identify CIE color valuesof known color samples or to identify, using a pre-trained neuralnetwork, the CIE color values of unknown color samples. In bothinstances an artificial neural network is employed to evaluate theextracted pixel data.

Artificial Neural Network

The processor 204 is configured by the program code 207, such as throughthe neural network training module 406 or neural network evaluationmodule 408 to evaluate the color values data extracted from the imagecaptured by the image capture device 102.

As a broad overview, artificial neural networks (ANN) are interconnectednetworks of nodes used to estimate or approximate functions that dependon a large number of inputs that are generally unknown. The outputsprovided by the neural networks are the results of a process ofiterative weighting and transforming values until an estimate of somecondition or final value(s) based on the input data is determined.

With reference to FIG. 5, the nodes of an ANN are organized into layersand are used to transform and weight input data from a source(s) (suchas pixel data corresponding to an image) according to one or morealgorithms. In a particular configuration, the neural network has three(3) or more layers. For example, a neural network implemented by theprocessor 204 has three (3) layers: an input layer, a hidden layer, andan output layer. In configurations where there are more than three (3)layers, it is typical to have two (2) or more hidden layers.

The source data provided to the input layer nodes are distributed tonodes located in the hidden layer. In one or more configurations, asingle input node provides an output value to one or more hidden layernodes. Thus, in a particular embodiment each hidden layer node isconfigured to accept multiple input signals. The nodes of the hiddenlayer combine input signals directed to a specific hidden layer node andcompare the combined (and weighted) values to an activation function inorder to determine an output signal. The output signal generated by eachnode in the one or more hidden layers is distributed to one of morenodes in the next layer of the network. Upon reaching the output layer,the output nodes are configured to accept multiple inputs and, throughthe use of an activation function, generate an output signal or value.

If the signal propagates from one node to another, then that path can bedefined and assigned initial weight values to transform the input intothe output. However, for an ANN to provide useful output values based oninput values, the entire ANN is trained so that for a given set ofinputs, an anticipated set of outputs is provided. Training an ANNincludes adjusting the weights provided for the paths connectingindividual nodes.

As provided herein the neural network is a software application storedin a memory location locally accessible by the processor 204.Alternatively, FIG. 2 provides (in dashed lines) that the neural networkis stored on a remote appliance 206 accessible by the processor 204. Forexample, the neural network is stored or hosted in a remote accessiblestorage device (e.g. cloud storage and hosting implementation) thatallows for dynamically allocated additional processors, hardware orother resources on an “as-needed” or elastic need basis. A database 208is in communication with the neural network appliance, whether local tothe processor 204 or hosted remotely. The database 208 is configured tostore data used to train the ANN.

Training the Artificial Neural Network to Recognize Colors

With respect to the step 306, the ANN is trained by using an image of asample having known CIE color values. By way of example, the ANN istrained using a collection of images taken of various sample colors,such as a fan deck of colors provided by a paint or color supplier. In aparticular embodiment, the image capture device captures an image of thecolor sample (e.g. a given color of the fan deck) along with thereference chart. In one arrangement, the ANN training procedure 306includes acquiring images of 206 different known CIE color valuessamples, along with the color reference chart, under at least three (3)different illuminants (D, H and S). In a further arrangement, multipleagents or image capture devices are used to capture the sample images.Thus, the input data provided during neural network training processincludes images of sample colors under different illuminants and takenusing different image capture devices. Through the use of differentcameras or image capture devices, the artificial neural network istrained to evaluate images taken with a number of different hardwareplatforms. In this way, variations in sensor response to colors andilluminants are presented to the neural network, thus providing a morerobust training set than if a single type of image capture device orilluminant were used.

The neural network training module 406 configures the processor 204 toextract the RGB color values from the sample and/or reference elementsof each image and store the RGB values in database 208 for eachilluminant. Thus, the neural network appliance uses data from threedatabases dbD, dbH and dbS each having 206 records. In order to ensurethat the known color values are correct, a commercial spectrophotometer(Datacolor 45G®) measures the reflectance of the 206 color samples inthe range of 400 nm to 700 nm with 10 nm interval. Based on thespectrophotometer measurements, CIELAB values (D65 Illumination, 10°Observer) are calculated for each color. The measured values arenormalized to the range of [0, 1] for ease of processing by the ANN.Those values are then used as the standard output for ANN training aswell as performance checking.

In one particular embodiment, the ANN has three layers: the input layerhas total 543 nodes, the hidden layer has total 69 nodes, and the outputlayer has total 3 nodes. The 543 input nodes can be divided into 2groups: Sam (sample) and Ref (reference) as provided in FIG. 5. The Samnodes include the RGB color values of the sample, and Ref includes theRGB color values of all the reference colors from the same image of theSam. The three (3) output nodes are called Opt (Output), and provide theoutput values of the ANN as CIE color values. In a particularimplementation, the CIE colors are CIELAB values. In the calculation ofthe ANN, a bias node can be added in both the input layer and the hiddenlayer, and a sigmoid function is used as the activation function at eachnode.

According to ANN training step 308, the processor 204 is configured bythe neural network training module 408, to select a record from thememory or database containing the RGB color values for the sample andreference elements, as well as the CIE color values (such as CIELABvalues) for a particular training image. The processor 204 provides thesample and reference element RGB color values to the input nodes of theANN, and sets the output node values to match the CIELAB values of theknown sample color. Without limitation to the internal workings ofvarious types and forms of ANN, by setting the output to the known CIEcolor (e.g. CIELAB) values, the weights connected the node layers willbe adjusted so as to correct the input values to determine the pre-setoutput value. Thus, where previously a known conversion ortransformation process was applied to each of the color values tocompensate for different illuminations or angles, the ANN uses machinelearning to adjust those weights and transformation functions withoutthe need of user input or control.

The training step 308 is repeated for each entry in the database orcolor records for a particular illuminant, and for each database ofilluminants. The processor 204 is further configured to validate theANN. To evaluate the performance of a given database, the processor 204using the validation module 410 selects a record in the database andinputs the sample and reference values as inputs. The ANN will outputcolor values based on the submitted inputs values as the result of itstraining. An output value check submodule of the validation moduleconfigures the processor to compare the output values of the ANN to theknown CIELAB values for that entry. Where the ANN has receivedsufficient training, the values of the Opt layer will substantiallymatch the CIE color (such as CIELAB) values of the same sample color inthe same record in the database.

Distributed ANN Training

In one or more further embodiments, the processor is configured topermit individual users of a distributed (remotely hosted) ANN tocollectively train the ANN. Here, individuals follow the steps 302-304provided. However, instead of training the ANN using a predeterminedilluminant database, each user takes an image of a known color sampleeither alone or with the reference color chart. As provided in step 310,individual users having access to a remotely accessible ANN applianceare able, through the crowd training module 416, to train the ANN usingcollections of images obtained on a variety of hardware platforms andlighting conditions. In this embodiment, the individual users are eachan agent of the multi-agent training system, such that each user's imagecapture device can have variations in hardware and software that willimpact the color values measured. Where a remote user has access to thecolor reference chart, the user captures an image of a sample havingknown CIE color values. The CIE color values, along with informationabout the image capture device and local processor(s) are collected andtransmitted to the ANN appliance. In this arrangement, the ANN is thustrained on different hardware conditions. As a result, the ANN can betrained for specific devices (e.g. iPhone) or lighting conditions (e.g.sunset) that are of particular interest to users. The data is providedas an input to the ANN and the known CIE color values are provided as apre-set output values. A distributed network is used to train the ANNusing a multitude of devices and illuminant conditions. By collectinglarge amount of crowd based training data, the chance for a particularuser to find the proper color in the standard database will increase,even if various noises are buried in the crowd based training data.

For a crowd based training dataset, once a large amount of informationhas been provided to the ANN and stored in one or more databases 208,the processor can provide additional input layers to the ANN based onspecific parameters, such as type of smartphone, or for a particulargroup of people, or for a particular geographic area, etc., to bettertrain the ANN and to provide more accurate color searching results forsaid parameters. In a further arrangement, the crowd training module 416configures the processor to generate new input nodes for the ANN inresponse to the type of data being submitted. For example, the processor204 is configured through one or more submodules to determine on thebasis of one or more hardware parameters, additional input nodes for usein the ANN. One or more input nodes indicating a particular make, modelor version of image capture hardware, software or a combination thereof,is used to train the ANN to be more accurate with respect to particulardevices.

In one particular embodiment, a database of known CIE color values (suchas database 108) is filtered for a specific hardware platform (e.g.iPhone® image capture devices). The results of the filtering arepresented to the ANN. Here the ANN is trained to identify CIE colorvalues from samples acquired from a specific hardware platform.

Evaluating Unknown Color Samples

Once trained, the ANN is used to identify the CIE color values ofsamples having unknown CIE color values. Since the ANN utilizesnonlinear activation functions, the output can be a nonlineartransformation of the input. Practically, the ANN appliance is wellsuited to color searching since color space is highly nonlinear and thecamera image sensor responses to color space is also not strictlylinear.

Returning to the flow chart of FIG. 3, if the CIE color values of asample captured in the image according to step 302-304 is unknown, thenthe processor 204 is configured by the neural network evaluation module412 to evaluate the RGB color values of the sample and referenceelements of the image under analysis and output proposed CIE colorvalues. The output CIE color values are used to search the colordatabases(s), such as by using a color search module 414 to identify thecolor having the highest match, which in most cases will be the truecolor of the sample under standard conditions irrespective of theuser-illuminant used during capture.

In one or more configurations, the user is able to select a hardware orsoftware profile prior to searching the color database using the ANN.For example, a graphical user interface is provided to a user indicatingone or more hardware profiles (e.g. make and model of smartphone) toselect from. By selecting a particular hardware profile, the processor204 is configured to access an ANN specifically trained on training datatailored to the selected hardware profile. Alternatively, the processoris configured to adjust or manipulate weights within the ANN in responseto the selection of a hardware profile.

General Information

While this specification contains many specific embodiment details,these should not be construed as limitations on the scope of anyembodiment or of what can be claimed, but rather as descriptions offeatures that can be specific to particular embodiments of particularembodiments. Certain features that are described in this specificationin the context of separate embodiments can also be implemented incombination in a single embodiment. Conversely, various features thatare described in the context of a single embodiment can also beimplemented in multiple embodiments separately or in any suitablesub-combination. Moreover, although features can be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination can be directed to asub-combination or variation of a sub-combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingcan be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising”, when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

It should be noted that use of ordinal terms such as “first,” “second,”“third,” etc., in the claims to modify a claim element does not byitself connote any priority, precedence, or order of one claim elementover another or the temporal order in which acts of a method areperformed, but are used merely as labels to distinguish one claimelement having a certain name from another element having a same name(but for use of the ordinal term) to distinguish the claim elements.Also, the phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including,” “comprising,” or “having,” “containing,” “involving,” andvariations thereof herein, is meant to encompass the items listedthereafter and equivalents thereof as well as additional items.

Particular embodiments of the subject matter described in thisspecification have been described. Other embodiments are within thescope of the following claims. For example, the actions recited in theclaims can be performed in a different order and still achieve desirableresults. As one example, the processes depicted in the accompanyingfigures do not necessarily require the particular order shown, orsequential order, to achieve desirable results. In certain embodiments,multitasking and parallel processing can be advantageous.

Publications and references to known registered marks representingvarious systems are cited throughout this application, the disclosuresof which are incorporated herein by reference. Citation of any abovepublications or documents is not intended as an admission that any ofthe foregoing is pertinent prior art, nor does it constitute anyadmission as to the contents or date of these publications or documents.All references cited herein are incorporated by reference to the sameextent as if each individual publication and references werespecifically and individually indicated to be incorporated by reference.

While the invention has been particularly shown and described withreference to a preferred embodiment thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade therein without departing from the spirit and scope of theinvention. As such, the invention is not defined by the discussion thatappears above, but rather is defined by the points that follow, therespective features recited in those points, and by equivalents of suchfeatures.

What is claimed is:
 1. A system for training a neural network to outputestimated CIE color values of a sample having unknown color values, thesystem comprising at least one image capture device, where the at leastone image capture device is configured to capture an image of at leastone color sample and a color reference chart comprised of known colorreference elements, the color reference chart including a plurality ofreference color element groups, wherein each of the plurality ofreference color element groups includes a matrix of color referenceelements, each color sample having known CIE color values, and whereinthe image capture device is configured to capture the image of the atleast one color sample and color reference chart under at least one of aplurality of known illuminants, each image capture device configured totransmit the captured image; a processor having memory configured byprogram code executed thereby to: receive the transmitted images; obtainthe known CIE color values corresponding to the color sample; extractfrom the transmitted images the RGB color values of the pixelscorresponding to area of the image depicting the sample and thereference color elements and train the neural network using theplurality of images and corresponding known CIE color values, wheretraining of the neural network includes: for each image, apply theextracted RGB color values for the sample and the reference colorelements as input nodes in an input layer of an artificial neuralnetwork and assign the known CIE color values for the sample color asthe output nodes of the artificial neural network.
 2. The neural networktraining system of claim 1, wherein the color reference groups arearranged to have fourfold rotational symmetry about the sample.
 3. Theneural network training system of claim 1, wherein the image capturedevice is configured to transmit the known color value of the samplealong with the captured image.
 4. The neural network training system ofclaim 1, wherein the processor is configured by code executing therebyto access the pre-determined color value from one or more storagedevices accessible to the processor and store, for each image, the RGBcolor values of the pixels corresponding to area of the image depictingthe sample and the reference color elements.
 5. The neural networktraining system of claim 1, wherein the processor is further configuredto: access from a storage location accessible by the processor, acollection of image data objects, where each image data object includesthe RGB color values of the pixels corresponding to area of the imagedepicting the sample, the reference color elements of a subset of theplurality of extracted RGB values for a given image, and the known CIEcolor values for the sample; evaluate the RGB color values as input nodevalues of the neural network; receive the output of the neural networkas CIE color values for color sample; compare the output CIE colorvalues to the known CIE color values stored in the image data object;and provide an alert to a user of the neural network where the known CIEvalues and the output CIE values are not substantially similar.
 6. Theneural network training system of claim 1, wherein each image device isconfigured to transmit a hardware profile to the processor along withthe captured image.
 7. The neural network training system of claim 6,wherein the training of the artificial neural network further includes:storing the captured image data and associated hardware profile in oneor more databases; filtering the entries of the database for imagescaptured with a particular hardware profile to obtain a hardwarespecific image dataset; applying the hardware specific image dataset asinput node values of the artificial neural network.
 8. The neuralnetwork training system of claim 6, wherein the hardware profile of atleast one image capture device differs from the hardware profile of atleast one other image capture device.
 9. The neural network trainingsystem of claim 1, wherein at least one image of the sample is capturedunder each of the plurality of illuminants.
 10. The neural networktraining system of claim 1, wherein each image capture device is remoteto the processor and configured to transmit the captured image via awireless communication channel.
 11. A system for training a neuralnetwork to output estimated CIE color values of a sample having unknowncolor values, the system comprising: at least one image capture device,where each image capture device is configured to capture an image of atleast one color sample and a color reference chart comprised of knowncolor reference elements, the color reference chart including one ormore reference color elements, the at least one color sample having aknown CIE color value, and wherein the image capture device isconfigured to capture the image of the at least one color sample andcolor reference chart under at least one of a plurality of knownilluminants, each image capture device configured to transmit thecaptured image; a processor having memory configured by program codeexecuted thereby to: receive the transmitted images; obtain the knownCIE color values corresponding to the color sample; extract from thetransmitted images the RGB color values of the pixels corresponding toarea of the image depicting the sample and the reference color elementsand train the neural network using the plurality of images andcorresponding known CIE color values, where training of the neuralnetwork includes: access from a storage location accessible by theprocessor, a collection of image data objects, where each image dataobject includes the RGB color values of the pixels corresponding to areaof the image depicting the sample, the reference color elements of asubset of the plurality of extracted RGB values for a given image, andthe known CIE color values for the sample; evaluate the RGB color valuesas input node values of the neural network; receive the output of theneural network as CIE color values for color sample; compare the outputCIE color values to the known CIE color values stored in the image dataobject; and provide an alert to a user of the neural network where theknown CIE values and the output CIE values are not substantiallysimilar.
 12. A system for training a neural network to output estimatedCIE color values of a sample having unknown color values, the systemcomprising at least one image capture device, where each image capturedevice is configured to capture an image of at least one color sampleand a color reference chart comprised of known color reference elements,the color reference chart including one or more reference colorelements, the at least one color sample having a known CIE color value,and wherein the image capture device is configured to capture the imageof the at least one color sample and color reference chart under atleast one of a plurality of known illuminants, each image capture deviceconfigured to transmit the captured image and a hardware profile of theimage capture device used to capture the image; a processor havingmemory configured by program code executed thereby to: receive thetransmitted images and hardware profile; obtain the known CIE colorvalues corresponding to the color sample; extract from the transmittedimages the RGB color values of the pixels corresponding to area of theimage depicting the sample and the reference color elements and trainthe neural network using the plurality of images and corresponding knownCIE color values, where training of the neural network includes: storingthe captured image data and associated hardware profile in one or moredatabases; filtering the entries of the database for images capturedwith a particular hardware profile to obtain a hardware specific imagedataset; applying the hardware specific image dataset as input nodevalues of the artificial neural network.